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ABSTRACT 

The past century has seen spectacular gains in the breadth and depth of medical knowledge, 
but the potential of these gains has been hampered by a slow system of disseminating 
knowledge. Over the course of medical education numerous technologies and methods have 
been used to deliver continuing medical education (CME) to health care professionals 
(HCPs). These methods have included postal correspondence, two-way radio conferencing, 
video conferencing and in the last decade the Internet. The emergence of the World Wide 
Web (WWW) in the early 1990s, coupled with increasing computer processing power, 
reduced computing costs and more creative content management systems have led to more 
CME materials and resources going 'online'. This has in turn greatly sped up the 
dissemination of medical knowledge. This paper reports on a contemporary study that 
assessed the pedagogical and instructional design (e-pedagogy) effectiveness of online CME 
courses offered by one large Australasian continuing education provider. 

Keywords: Web-based learning; continuing medical education (CME); educational 
technology; distance learning; e-learning; World Wide Web 


INTRODUCTION 

Continuing medical education (CME) for health care professionals (HCPs) is a vital 
requirement for practitioner development and registration (Steinman, Landefeld & Baron 
2012). It is also of strategic importance for improving the health of patients and communities. 
Add to this, health care recipients (patients) and legislators are demanding more clinical and 
fiscal accountability for the type and cost of care that is provided. This necessitates HCPs to 
keep their knowledge of contemporary practices up-to-date (Todd & Bonnie 2011). Over the 
years, various learning technologies and methods have been applied to address the CME 
requirements of health care professionals. This has included mail correspondence courses, 
audio teleconferencing, face-to-face lecture style meetings, and video conferencing. The 
emergence and rapid growth of the World Wide Web, and Internet based content 
management systems have introduced new opportunities for providing CME. 

Internet based technologies permit the widespread distribution of digital content to many 
users simultaneously anytime and anywhere. For busy HCPs the ability to study when they 
can or need the information is highly desirable. This convenience factor coupled with the 
benefits that come from using a content management system such as standardising course 
content and delivery, automated tracking and reporting of learners' activities lessen the 
administrative burden on CME providers. Moreover, internet based content management 
systems can be designed to include outcomes assessment to determine whether learning 
has occurred and direct learners to other areas for further information of clarification if 
needed. These advantages in learning enhancement are perhaps less well recognised but 
potentially more revolutionary to those related to learning delivery. For HCPs learning online 
offer CME providers and accreditation agencies a new paradigm based on adult learning 
theory, which states that adults learn by relating new learning to past experiences, by linking 
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learning to specific needs, and by practically applying learning to real world clinical 
parameters, resulting in more effective and efficient learning experiences. 

The primary purpose of CME is to help keep practitioners up-to-date with content knowledge 
and procedures related to areas of practice. The introduction of online CME via the Internet 
provides opportunities to address the challenges faced by both HCPs and CME providers. 
For HCPs, the Internet provides an effective way to access the latest information, technology, 
and procedures in a place and time that is of convenience to them, and when they require it- 
in essence just-in-time learning. Internet based technology also offers educational providers 
another delivery format option to provide a service to their students. The evolution of online 
CME is reliant on the ongoing evaluation of the learners’ needs and experiences. In effect 
online CME offers the potential to affect how HCPs learn, whilst improving the quality of that 
learning through peer-assisted learning and mass-dissemination of a shared knowledge 
base. To that end the purpose of this study was to evaluate both the process and outcomes 
of 42 online CME courses offered to HCPs in Australasia by a private industry accredited 
educational provider. 

Medical education is not unlike other educational fields for other disciplines in that the 
purpose or challenge of CME lies in practitioners accessing timely and appropriate course 
material. This material is prepared by accredited peers and is based upon sound industry 
and current legislative requirements and best practice methods (Farrow, Gillgrass, 
Pearlstone, Torr & Pozniak 2012). This process of CME quality assurance relies on 
practitioner accreditation and regulatory bodies determining appropriate content and 
evaluative processes which courseware providers can then use to build their courses around. 
In Australia CME for general practitioners and physicians is regulated by the Royal Australian 
College of General Practitioners (RACGP) and the Royal Australian College of Physicians 
(RACP). Both of these bodies had approved the 42 CME courses offered by the provider 
utilized in this study. 

The 42 CME courses examined in the study could be defined as asynchronous distance 
learning as the education took place at both a time and location distant to the teacher or 
providing institution (Phipps & Merisotis 1999). Over time there have been many evaluations 
of the effectiveness of distance learning programs, which concluded that effective learning 
can occur via distance learning (Abrami, Bernard, Bures, Borokhovski & Tamim, 2011; 
Banda, Franco, Simpson, Brennan, McKanry & Bragg 2012). These reports do, however, 
indicate that the methodological quality of such research studies are often poor, indicating 
that the results should be interpreted with care (Barnes, 1998, Pullen, 2012). As a result of 
these studies, health care professionals and continuing medical education providers are 
questioning the value and/or effectiveness of some forms of education, particularly in the 
complex world of information, communication technology supported learning (or e-learning). 


RESEARCH QUESTIONS 

This study was designed to provide a better understanding of HCPs' experiences learning 
online and to determine the effectiveness of this technology as a learning medium for CME. 
Therefore, this study explored two broad research questions--first to quantify the overall 
effectiveness of the web-based CME instruction. Through this question answers were sought 
to will learners have a positive experience; learn from the course and apply that knowledge 
into their clinical practice. Research question 2 sought answers to assessing the influence of 
several factors (moderators) that were predicted to influence the effectiveness of the online 
learning event. The moderators were particular to the courses examined and related to the 
pedagogical and instructional design characteristics embedded in the online courses, as 
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indicated in Table 1. 

Table 1: Moderator and hypothesis examined 


Moderator - description _ 

Moderator 1 -Web-based education will be effective when feedback is provided. _ 

Moderator 2 -Web-based education will be effective when immediate feedback is provided. 
Moderator 3 - Web-based education will be effective when the course allows the learner 

the opportunity to apply the new knowledge to clinical or question-type responses. _ 

Moderator 4 - Learner ability to break lesson material down into learner-managed 

components will be viewed favorably by the learners. _ 

Moderator 5 - Learner opportunity to apply course content into practice has been found to 
be beneficial in face-to-face education and, for the purposes of this study, the opportunity 
to apply course content to practice or to actively learn will be positively viewed by learners. 
Moderator 6 - Learner opportunity for autonomy or self-pacing of course material has been 
found to be beneficial in the classroom setting and this study examined whether self- 
pacing is beneficial to Web-based learning. _ 


THE EDUCATIONAL PROVIDER AND CONTENT 

The educational provider began delivering CME to HCPs in 1994 and in 2002 commenced 
offering CME courses online. The educational content delivered by this provider is divided 
into 25 therapeutic areas, for example nutrition, pediatrics and mental health to name just 3 
areas. The 25 therapeutic areas combined provide over 150 hours of continuing accredited 
medical education. The provider currently has over 39,000 registered users, of which just 
over 8,000 were Doctors (family doctors or general practitioners), enrolled in at least one 
their online CME course. All courses could be considered to be asynchronous in that any 
direct human feedback was delayed, with interactive feedback only being available to those 
courses which contained multiple choice type question and answers. 


MATERIALS AND METHOD 

The study used both qualitative and quantitative techniques to describe and analyse learners' 
attitudes towards web-based instruction, from the perspective of effective instructional and 
pedagogical design. It also examined the interaction and instructional methods, strategies 
and activities associated within each web-based CME course. The research questions- 
methods, strategies and activities-were clustered into four major themes, three of which 
were centered on Kirkpatrick's (1987, 1996, 1998) multilevel assessment criteria: (Level I) 
learners' reactions to learning, (Level II) achievement, and (Level III) application into practice. 
The fourth area (level IV) investigated instructional design and pedagogical influences on the 
other three levels. The underlying objective of the study was to examine if HCPs had a 
positive reaction to and acquired knowledge from the web-based CME learning experience. 
In addition, the study sought to identify effective and efficient online pedagogical strategies 
which were utilized in the online course design. 

Instructional effectiveness was evaluated by several criteria, including learning (cognitive) 
achievement (Level II), participant satisfaction with instructional courseware (Level I), 
retrospective self-reported performance change (Level III), and instructional transitions 
occurring through the use of the courseware. The following electronic quantitative evaluation 
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instruments and qualitative methods were used for collecting data: 

1. Demographic Profile Survey (DPS) - collected data on demographics, computer 
experience, computer access, and computer usage by participants. The DPS was a 
subset of the Courseware Evaluation Survey (CES) (Level I) and survey data were 
tallied using descriptive statistics. 

2. Pre-test and post-test (learner achievement --Level II) - evaluation of the pre- and 
post-test scores allowed for comparison of learning outcomes and were assessed 
using a paired t-test as an objective measure of achievement for the material. 

3. Post-learning Performance Self-Assessment Survey (Level III) - participants self- 
reported if and how they had used course knowledge and materials in their clinical 
practice. This was a retrospective question and results were tallied using descriptive 
statistics. In addition 24 of the 42 examined courses used material, such as diagnostic 
screening tools or differential diagnostic charts, for participants to take away and use 
with their patients in the clinical setting. This material was termed clinical tools. The 
self-reported behaviour change of participants undertaking courses which used either a 
course with a clinical tool or a course without a clinical tool was measured using 
correlation statistics. 

4. Courseware Evaluation Survey (CES-Level I) was designed to collect information on 
the participants' perceptions of the quality and effectiveness of the Internet delivered 
course. The survey included 39 items, distributed among 8 evaluative categories that 
were derived from the instructional design and pedagogical characteristics of the 
learning site and supporting literature review. The categories included content, 
navigation and organisation, overall impressions, media utilisation and learning style. 
Each category included 1 to 18 evaluative statements, which participants were asked 
to rate on a Likert scale from strongly positive to strongly negative. All statements were 
positively worded. The survey also included one open-ended item to elicit additional 
comments or suggestions. The reliability of the 39 Likert type items was determined by 
calculating the inter-correlations (Cronbach's alpha) for those statements, refer to 
Table 2, which gave a mean inter-correlations value of 190.9 with an estimated 
Cronbach alpha reliability of 0.92. According to Nunnally 1 (1978), this was significant; 
therefore, the CES was considered statistically very reliable. 
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Table 2: Level I inter-correlations and Cronbach alpha reliability 


RELIABILITY ANALYSIS -SCALE (ALPHA) 


N of Cases = 168.0 
N of 


Statistics for 

Mean 

Variance 

Std Dev 

Variables 


Scale 

190.8571 

343.4645 

18.5328 

39 



Item Means 

Mean 

Minimum 

Maximum 

Range 

Max/Min 

Variance 


4.8938 

2.4821 

5.7619 

3.2798 

2.3213 

.7050 

Item Variances 

Mean 

Minimum 

Maximum 

Range 

Max/Min 

Variance 

Inter-item 

.8941 

.1825 

3.7565 

3.5740 

20.5844 

.7480 

Covariances 

Mean 

Minimum 

Maximum 

Range 

Max/Min 

Variance 

Inter-item 

.2082 

-.2964 

2.0657 

2.3621 

-6.9690 

.0346 

Correlations 

Mean 

Minimum 

Maximum 

Range 

Max/Min 

Variance 


.3249 

-.1298 

.8936 

1.0234 

-6.8865 

.026 


Analysis of Variance 


Source of Variation 

Sum of Sq. 

DF 

Mean Square 

Q 

Prob. 

Between People 

1470.7326 

167 

8.8068 



Within People 

8853.3333 

6384 

1.3868 



Between Measures 

4500.6612 

39 

118.4385 

3245.3563 

.0000 

Residual 

4352.6722 

6346 

.6859 



Total 

10324.0659 

6551 

1.5760 




Grand Mean 4.8938 
Reliability Coefficients 39 items 

Alpha = .9221 Standardized item alpha = .9494 SPSS V 16.0 


PARTICIPANTS 

Participants were all health care professionals enrolled in one of the forty-two online CME 
courses. The number of responses varied from survey to survey. The reason for this variance 
was due to the self-pace nature of the courses. Participants could complete course material 
over a matter of hours or over a period of several months. The results reported in this paper 
occurred over a 4-month period in which 168 participants completed the Level I survey, 313 
participants completed the Level II survey, and 330 participants completed the Level III 
survey. 


COURSE SELECTION AND PROGRESSION 

All participants once enrolled in the course of their choosing received an emailed password 
to use to open and engage with the course instructions, learning objectives and content 
within a purpose made content management system (CMS), which interface wise was very 
similar to other CMS such as Blackboard. Once in the CMS participants' first task was to 
introduce themselves to the course facilitator and fellow participants through an online 
discussion board. Participants then undertook the Level II survey which was a pre-test of 
their subject knowledge for the course that they had enrolled in. Pre-test knowledge tests 
were closed question type responses in which participants chose between multiple-choice 
responses. Pre and post-tests were consistent within each course. Once pre-testing was 
completed participants were able to proceed into the course content. The learning material 
covered clinical theory and best practice case studies and descriptors, including treatment 
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and prognostic options. Each course utilized video and clinical file notes to discuss the theme 
area being studied. From the 42 available courses 24 used diagnostic screening and testing 
tools (patient survey forms; diagnostic assessment proformas/flow charts etc. that were 
designed to be used with or given to patients by their doctor to explain their condition and 
treatment option/plans) as a specific teaching component of the course. These clinical tools 
were designed to assist participants with the course material and best practice approaches 
being studied whilst online and when dealing with patients in the real-world clinical setting. 

Following completion of the course participants completed a post-test of knowledge (Level II). 
This information was then used in conjunction with the pre-test scores to provide a 
comparison of learning outcomes, or learning that was derived from the course. Following 
completion of the post-test participants completed a course evaluation survey (CES Level I) 
which asked them to comment from strongly agree to strongly disagree on 8 categories 
which ranged from instructional design characteristics through to course content and learning 
material. Incorporated in the CES were questions that asked the participants if and how they 
had used the knowledge gained from the course and clinical tools, for the 24 courses that 
incorporated such tools, in their professional clinical practice. This information was used to 
address the evaluation Level III criteria of application of knowledge and bahaviour changes 
into practice. Data obtained from this component of the survey were reported descriptively. 


RESULTS 

A timed convenience sample of responses was chosen from the CME provider, which 
provided online continuing medical education courses. The timed convenience sample 
occurred over a period of 4 months. During this period, 1428 pre-tests were undertaken, 
which represented 789 individual users, as some participants could and were enrolled in 
more than one course at the same time. During the same time period, 703 post-tests were 
completed, representing 330 individual users. From the initial 789 individual pre-test 
participants, 40% (n = 313) completed both the pre-test and post-test (Level II). The Level I 
participant satisfaction survey was completed by 51% (n = 168 of the individual 330 
participants) of individual post-test participants. The Level III survey was completed by 100% 
of the individual post-test participants (n = 330). As previously indicated, the participants 
could choose to complete the course at their own pace, and this extended to the survey tools 
which could be completed at the key learning course milestones or during their next logged- 
on session. All the survey tools, apart from the Level I course evaluation survey, were 
required to be completed before the participant could progress-to the next learning area. 

The first research question was to quantify the effectiveness of the web-based CME courses 
from the perspective and achievement of the learners. This objective was concerned with 
ascertaining whether students enrolled in the online educational environment acquired 
knowledge (Level II), applied that course knowledge into their professional practice (Level III) 
and, in general, quantified their reactions to studying in an online environment (Level I). 
Whilst the evaluation criteria levels go in numerical order from Level I through to Level III, the 
surveys were not implemented in that order and so will not be discussed in numerical order. 
Instead, they will be discussed in the order the participants encountered them in their 
courses. 

Learner achievement (Level II) data revealed whether learners enrolled in the web-based 
CME courses acquired the knowledge disseminated through the educational experience 
(Level II). Table 3 shows pre- and post-test differences indicating that on average 
participants scored significantly higher on the post-tests. Upon completion of the post-tests, 
participants were asked to complete the CES (Level I); DPS and Level III application of 
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content and material into practice - as measured by self-reported behaviour and practice 
changes following the studying of the course. 


Table 3: Summary of means for pre-and post-course test of knowledge 


Learning 

module 

Pre-test 

mean 

SD 

Post-test 

mean 

SD 

f-test 

P 

n = 42 

78.5 

18.8 

94.5 

8.6 

-5.2 

< 0.0005 


Application of knowledge and behaviour change into practice (Level III) 

Answers to the questions about self-reported behaviour or practice change together with self- 
reported clinical assessment tool use, for those courses that incorporated the use of a clinical 
tool (n = 24) were asked of participants following completion of the course, this is presented 
in Table 4. 


Table 4: Practice behavior change with and without tool use 


Behaviour % (n=303) 

Practice 

unchanged 

Confirmed 

practice 

Reviewed 

practice 

Modified or changed 
practice 

Learning module Total 

3% 

28% 

39% 

30% 

Without tool (n=18) 

5% 

34% 

36% 

25% 

With tool (n=24) 

2% 

24% 

39% 

33% 

Reasons for and 
against tool use in 
practice 

Not 

interested/helpful 

Maybe use 

Probably use 

Definitely use or are 
using 


3% 

4% 

37% 

56% 


An examination of the correlation between behaviour responses and tool usage was 
investigated using Pearson product-moment correlation coefficient. A preliminary analysis 
was performed to ensure that no violation of the assumptions of normality, linearity and 
homoscedasticity etcetera occurred. Findings revealed a Pearson correlation of 0.750 at the 
0.01 level of significance, indicating that learning did occur from the commencement of the 
course to the completion of the course. Furthermore, as shown in Table 4, those courses 
which utilised a clinical or diagnostic tool as a key component of the learning within the 
course achieved a higher level of behavioural modification as compared to courses that did 
not utilise a clinical tool. Further validation of the association between behaviour change and 
courses which used a clinical tool were obtained from examining the statistical significance of 
the difference between the two correlation coefficients, behaviour change and tool usage, 
subsequent Z obs 2 analysis revealed a Z D b S value of 3.8 which indicated that those courses that 
utilised a clinical tool as component of the course learning achieved better results, or 
changes in participant behaviour, than those courses which did not utilise a tool. 


Learner satisfaction (Level I) and Demographic Profile Survey (DPS) 

A post-course, demographic survey (conducted at the completion of the course at the 
request of the provider) and a learner satisfaction survey were used to investigate research 
question 1 -quantify the overall effectiveness of the web-based CME instruction. The 
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demographic characteristics, as reported in Figures 1 and 2 indicate that participants were 
evenly distributed between age, gender and self-reported IT skills, with general practitioners 
(family doctors) being the largest group of users. 










20-30 31-40 41-50 51+ GP Other 


Age range Occupation 

Descriptors - Age range and Occupation 


Figure 1: Demographic characteristics of participants 
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Male 


Female 


Total 


Internet 

competency 


Number of online 
Computer courses previously 

competency undertaken 


Figure 2: Self-reported information technology proficiency and previous online CME 
experience 


In relation to participant satisfaction experiences with the courses studied 87% of participants 
agreed with positive statements regarding the instructional design-course structure 
(descriptive categories 1,3,5-7 in Table 5)—and learning effectiveness-content (95%, 
descriptive category 2) and supporting a problem based learning style (73%)—of the courses 
examined. Participants general experience with online CME, are shown in Table 5 indicates 
that overall participants felt engaged with the CMS and that they could learn using this 
medium. 
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Table 5: Level I Course Evaluation Survey (CES) - percentage of participants strongly 
agreeing or agreeing with issues relating to the 8 descriptive categories and 39 Likert type 
responses 


Descriptor 

Percentage of 
participants strongly 
agreeing/agreeing' 

1. Course structure - 

96 

* Objectives clear 

93 

* Organisation clear/appropriate 

99 

2. Content - 

95 

* Content appropriate to my profession 

99 

* Relevant to my practice 

97 

* Based on Problem Based Learning (PBL) 

96 

* Builds on my current knowledge level 

94 

* Builds on my experience level 

90 

3. Graphics/media - 

76 

* Purposeful 

66 

* Appealing presentation 

79 

* Well organised 

84 

4. Medium - 


* Most effective 

Text 75 

* Least effective 

Hyperlinks 49 

5. Instructional design - 

91 

* Rate of course progression appropriate 

97 

* Pacing - allowed me to progress at own pace 

96 

* Interactivity of material appropriate 

82 

* Organisation of material appropriate 

92 

* Easy navigation through site/material 

95 

* Purposeful design 

83 

6. Home page - 

97 

* Organisation well designed 

98 

* Instructions easy to follow 

96 

7. Impressions - 

73 

* Future participation 

100 

* Effective way of learning 

97 

* Appealing way to learn 

92 

* Favorable way of learning 

97 

* Easy to use 

97 

* Increased my confidence in using this learning medium 

73 

* Feedback from providers appropriate to my queries 

59 

* Above feedback timely 

55 

* Interaction with others improved the course 

69 

* Interaction with others was appropriate 

66 

* Interaction with others timely 

54 

* Course offered value for money 

83 

* More effective than face-to-face courses 

54 
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Descriptor 

Percentage of 
participants strongly 
agreeing/agreeing' 

* Very usable format for learning 

65 

* Course layout appealing 

54 

* Course content of high academic standard 

61 

* Learning objectives appropriate 

56 

8. Learning style supported 


The course made me adopt a: 


* Problem based learning style 

73 

* Strategic learning style 

24 

* Surface learning style 

3 

* Would you choose to learn online again? 

100 Yes 


FINDINGS 

Most of the participants (85%) in this study had undertaken an online CME course prior to 
undertaking the current one. Over 96% of the participants reported that their Internet skills 
were good to excellent and that the majority (93%) had good to excellent computer skills. 
This finding may support the assumption that having previous good experiences learning 
online supports future studies online; it may also indicate a preference for those with good 
computer skills to enroll in courses which utilize this technology. 

Health care professionals in this study did learn from this educational format as indicated by 
pre and post test scores. Additionally, the vast majority (95% - as indicated by descriptive 
category 2 findings) agreed that their knowledge of the course area increased. The course 
material was useful and applicable to their clinical practice, and was conducive to higher 
order learning (73% indicated the online course supported their adoption of a problem based 
learning style). In addition, courses that incorporated a clinical or diagnostic tool as key 
component of their online learning program increased participants practice changes more so 
than those courses which did not utilize a similar learning tool (Z obs value of 3.8). This 
indicates that the use of such clinical and diagnostic tools improves the learners’ ability to 
learn and apply that new knowledge to their clinical practice. In relation to the moderators 
and their anticipated effect on participant learning the study found that all six moderators 
were supportive of student learning, as shown in Table 6. 

Table 6: Instructional and pedagogical design influences on student learning and satisfaction 


Moderator - description 

% of participants 
Strongly agreeing / 
Agreeing with 
moderator (n=168) 

Moderator 1 -Web-based education will be effective when 
feedback is provided. 

74% 

Moderator 2 -Web-based education will be effective when 
immediate feedback is provided. 

95% 

Moderator 3 - Web-based education will be effective when the 
course allows the learner the opportunity to apply the new 

99% 
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knowledge to clinical or question-type responses. 


Moderator 4 - Learner ability to break lesson material down into 
learner-managed components will be viewed favorably by the 
learners. 

97% 

Moderator 5 - Learner opportunity to apply course content into 
practice has been found to be beneficial in face-to-face education 
and, for the purposes of this study, the opportunity to apply course 
content to practice or to actively learn will be positively viewed by 
learners. 

97% 

Moderator 6 - Learner opportunity for autonomy or self-pacing of 
course material has been found to be beneficial in the classroom 
setting and this study examined whether self-pacing is beneficial to 
Web-based learning. 

96% 


These findings were consistent across all 42 courses examined in this study. The continuing 
medical education provider utilised the same instructional design and delivery methodology 
across all of its courses offering participants consistency of approach but also a sense of 
familiarity for those who had previously studied with this provider. Such an approach ensured 
that instructional design and pedagogy were kept consistent between courses, despite their 
differing coverage of content or therapeutic areas. The information gained from this study is 
helpful, but limitations do exist which are associated with case studies and the small number 
of participants who undertook the courses during the data collection period. However, this 
study was effective in evaluating personal experiences of the study group with online CME 
and provides insight into helpful instructional design and online pedagogical approaches. 

The development, use, and evaluation of online CME for health care professionals, and other 
adult learners, is essentially still in its infancy, and many options exist for future evaluation 
studies. The primary purpose of the current study was to assist the educational provider to 
ascertain how successful they have been in their instructional and pedagogical course 
design. To this end they have largely ensured that learners were indeed learning content 
knowledge through their online courses. However what was surprising is that in an online 
environment the use of hyperlinks within the courses was not viewed as being positive. 
Perhaps the inference here is that learners would prefer to have access to all of the content 
in one place rather than being redirected to other areas of the page or to other web pages. 
This finding has implications for how future courses could be constructed and taught. Whilst 
this evaluation study assisted the educational provider with assessing their own online 
courses, similar evaluations using larger, random samples would allow for greater 
generalizations to be made about the Internet and content management systems for online 
CME and learning in general. 


ENDNOTES 

Nunnally recommended a minimum level of 0.7; reliability values range from 0 to 1, with higher 
values indicating greater reliability. 

2 

“Behaviour (no tool)”, r-value = 0.863 (r1), “behaviour with tool” r-value =0.750 (r 2). 
Transformation of r to Z values as indicated by Cohen (1988) and McCall (1990) gives rl is Z1 = 
1.313, r2 is Z2 = 0.973 and n values as ascertained were n 1 = 205 and n2 = 306 respectively. 
The transformation of r-values into a standard score value (Z values) is done to ensure that the 
sampling distributions are approximately normal (Cohen, 1988; McCall, 1990). The formula then 
used was: 
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Zofa = Zl-Z2/\1/(JV1-3) + l/(JV2-3) 


The number in bold is the overall percentage of participants agreeing with the questions covered 
by the descriptor. 
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